Skip to main content

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: nick.clark Date: Jul 19, 2013 10:09am
Forum: web Subject: Re: Why can't I access old crawled sites who have recent robot.txt exclusions?

I am wondering the same thing. Is it possible that it's just a waiting game to see a new robots.txt file? I was in the process of rebuilding a site and normally I turn off robots while I am building. How soon does the IA scan the domain to put the archive back online?